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@ Networked system for display of multimedia presentations. 



@ Disclosed is a networiced multimedia infor- 
mation system (10) which may be utilized to 
record, store and distribute multimedia pres- 
entation (35,40,60,70) togettier with any supple- 
mental materials that may be referenced during 
tiie presentation. The recorded presentation, 
together with the assodated supplemental 
materials, may. be simultaneously presented on 
a display (510) containing two separate viewing 
windows. The effects of network (20) conges- 
tion are minimized by prefetching audio and 
video data for storage In audio and video buf- 
fers (110,115). An adaptive control algorithm 
compensates for network congestion by 
dynamically varying the rate at which video 
frames are retrieved over the networic (20), in 
response to networic traffic conditions. The 
audio playback speed is reduced if the audio 
data does not arrive fast enough over the net- 
work to maintain the desired size of the audio 
buffer after the amount of video data transmit- 
ted aaoss the networic has been reduced to a 
minimum value. 
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1 EPO 

FIELD OF THE INVENTION 

The present invention relates to a networked mul- 
timedia Information system, and more particularly, to 
a systemfor the storage and distribution of a recorded 
multimedia presentation, such as a seminar or confer- 
ence, together with any supplemental materials, such 
as viewgraphs, slides or handouts, that are refer-, 
enced during the presentation. 

BACKGROUND OF THE INVENTION 

Businesses today require access to ever- 
Increasing-jamounts of infomration. such as reports, 
conrespondence, contracts, engineering drawings 
:i and marketing materials. Making this infonmation 
available to a large number of distributed corporate 
employees is often a difficult problem. 

There are a number of connmercially available 
document management systems, such as NCR's 
Document Management System and Bellcore's Su- 
• perBook Document Browser, which allow still Images 
of documents, photographs and diagrams to be elec- 
tronically stored for subsequent access over a net- 
work. 

For example, NCR's Document Management 
System controls the flow of image documents over a 
corporate network. A user employing a microcomput- 
er or workstation can access documents that have 
been stored in an electronic forrtiat on a central file 
server using imaging technology. 

However, there are few commercially available 
document management systems that also allow con- 
tinuous media, such as recordings of audio and video 
presentations, to be stored and distributed over a net- 
work. Although multimedia applications make enor- 
mous demands on the resources of a computer sys- 
tem, recent advances in storage, processing, conv 
pression and network technologies have facilitated 
the development of networked multimedia infonma- 
tion systems. These technologies have advanced to 
where it is now possible to combine numerous media, 
including documents, audio and video, into a single 
system. V 

Since the number of employees who may partici- 
pate in a corporate presentation, such as a confer- 
ence or a seminar. Is often limited by scheduling and 
location constraints. It is desirable to provide a sys- 
tem that can record and store corporate presentations 
for subsequent viewing by additional corporate em- 
ployees at convenient times arid locatk>ns. 

A major difficulty with subsequent viewing of re- 
corded presentations, however, is the poor reproduc- 
tion of supplemental materials associated with the re- 
corded presentation, such as viewgraphs, slides and 
handouts. Accordingly, a need exists for an improved 
networked multimedia system capable of storing and 
Indexing multimedia presentations, such as seminars 
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and conferences, for subsequent access over a com- 
munications network. A further need exists for a net- 
worked multimedia system that separately processes 
the supplemental materials, such as viewgraphs, 

5 slides and handouts, associated with a recorded pre- 
sentation in order to improve the clarity of the repro- 
\ duced supplemental materials. In addition, a need ex- 
ists for a networked multimedia system that is capa- 
ble of synchronizing the audio and video components 

10 of a recorded presentation with the display of any sup- 
plemental materials referenced during the presenta- 
tion. 

SUMMARY OF THE INVENTION 

IS 

Generally, according to one aspect of the Inven- 
tion, a networked multimedia infonmation system is 
utilized to store and distribute multimedia objects, 
such as recorded presentations, over a network to a 

20 plurality of workstations. The networked multimedia 
infonmation system includes an information retrieval 
system, a still Image storage and retrieval system and 
a continuous media storage and retrieval system. 
According to one aspect of the Invention, the net- 

25 worked multimedia system is utilized to record, store 
and distribute multimedia presentations, such as 
seminars, meetings or conferences, and any supple- 
mental materials thatmay be referenced during the 
presentation, such as viewgraphs^ slides, blackboard 

30 annotations or handouts. 

According to another aspect of the invention, the 
recorded presentation, and the associated supple- 
mental materials may be simultaneously presented 
on a display containing two separate viewing win- 

35 dows. Preferably, the presentation of the audio and 
video components of the recorded presentation is 
synchronized with the presentation of the supplemen- 
tal materials. In one embodiment, the synchronteation 
of the recorded presentation and the associated sup- 

40 plemental materials is accomplished by means of hy- 
perlink files, which Identify the supplemental materi- 
als and the conresponding frame number In which the 
supplemental materials are referenced. 

According to a further feature of the Invention, a 

45 user can manipulate the supplemental materials dis- 
played in one viewing window to control the video 
playback in the second viewing window, thereby al- 
lowing the supplemental materials to serve as an In- 
dex to the recorded presentation. Once the user has 

so selected the desired page or portion of the supple- 
mental materials, the recorded presentation can be 
restarted from the video frame where the selected 
page or portion of the supplemental materials is ref- 
erenced in the recorded presentation. 

55 Anotherfeature of the invention minimizes the ef- 
fects of network congestion by prefetching audio and 
video data for storage in audio and video buffers. In 
one embodiment, the networked multimedia system 
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compensates for network congestion by using an 
adaptive control algorithm, which dynamically varies 
the rate at which video frames are retrieved from the 
respective file server over the network, In response 
to network traffic conditions. In order to maximize the 
playback quality of the recorded presentation, the au- 
dio cortiponentof the recorded presentation is given 
preference over the video component 

Thus, when network congestion conditions are 
extrerhe, the network multimedia system will transmit 
only audio data, without any video data, to the respec- 
tive workstetion. If the audio data does not arrive fast 
enough over the network to maintain the desired size 
of the audio buffer when there Is no video data being 
transmitted, the network multimedia system will re- 
duce the speed at which the audio data Is played by 
the workstetion until the amount of audio data In the 
audio buffer has returned to the desired size. 

A more complete understanding of the present in- 
vention may be had by reference to the following De- 
tailed Descriptton with the accompanying drawings. 

BRIEF DESCRIPTION OF DRAWINGS 

FIG. 1 1s a schematic block diagram illustrating a 
networked multimedia Infonmatlon system ac- 
cording to the present Invention; 
FIG. 2 Is a schematic block diagram Illustrating a 
still image Information capture subsystem; 
FIG. 3 Is a schematic block diagram Illustrating a 
continuous media information capture subsys- 
tem; 

FIG. 4 Illustrates an object profile that maintains 
bibliographic Information for a multimedia object 
stored in the networked multimedia system of 
FIG.1; 

FIG. 5 Illustrates a workstetion display having 
two windows for simultaneous display of a re- 
corded presentation, together with its associated 
supplementel materials; 

FIG. 6 Illustrates a still Image hyperlink file that 
maintains the storting video frame number asso- 
ciated with each page of the still Image supple- 
■ mentel materials; 
FIG. 7 Illustrates a flow chart describing an exenrv 
plary still image synchronfeation subroutine ac- 
cording to the present Invention as utilized by a 
knovyn video process to synchrontee the display 
of the recorded presentation with its still image 
supplemental materials on the workstetion dis- 
play of FIG. 6; 

FIG. 8 Illustrates a continuous media hyperlink 
file that maintains an event marker and the asso- 
ciated starting video frame -number for each 
blackboard event, or stroke, that is placed on an 
electronic blackboard; 

FIG. 9 Illustrates a flow chart describing an exenv 
plary continuous media synchronization subrou- 



tine according to the present invention as utilized 
by a known video process to synchronize the dis- 
play of the recorded presentation with Its contin- 
uous media supplemental materials on the work- 
5 station display of FIG. 5; and 

FIG. 10 illustrates a flow chart describing an ex- 
. emplary data buffer monitoring subroutine ao- 
; cording to the present Inventfon as ijtilized to 
. maintain a pre-defined amount of data In audio 
10 and video buffers. 

DETAILED DESCRIPTION 

The networked multimedia system 10 according 
15 to the present invention Is Illustrated In FIG. 1. The 
multimedia system 10 stores and Indexes multimedia 
objects, such as documents, diagrams, photographs, 
audio, video and animation, for access by a user over 
a network 20. 

20 A user employs a workstation, such as worksta- 
tion 15. which may be a Sun SPARC workstetton, 
model 10SX. 10M or classic M, produced by Sun Mi- 
crosystems, Inc., of Mountain View, CA.. or a similar 
workstation offering high-resolution Imaging and sup- 

25 port for real-time audio. Each workstation, such as 
the workstation 15, may be comprised of a display 
510, a processing unit 16, a memory unit 18. buses 
22, audio hardware 23, an audk) buffer 110, a video 
buffer 115 and a user interface 24. such as a mouse 

30 and/orakeyboard.Apreferredembodlmentofthedis- 
play 510 is discussed below relative to FIG. 5. 

The network 20 may be a local area network 
(LAN) or a wide area network (WAN), such as a na- 
tionwide corporate Internet consisting of Ethernet 

35 networks Interconnected by leased data circuits. Al- 
. ternatively, the network 20 may be embodied as; e.g., 
an asynchronous transfer mode (ATM) network or an 
Integrated Services Digital Network (ISDN). As dis- 
cussed further below, to accommodate the low band- 

40 width which is typical of these networks, data files are 
preferably compressed prior to transmission over the 
network 20. 

As Illustrated in FIG. 1 and discussed further be- 
low, the networked multimedia system 10 utilizes an 

45 Information retrieval system 50 to coordinate the ac- 
cessing of multimedia objects stored by a plurality of 
file servers, such as file servers 35, 40. The Infonma- 
tlon retrieval system 50 will interact with a still image 
storage and retrieval system 60 to retrieve the still Im- 

50 age objects stored by an associated file server 35. In 
addition, the infonnation retrieval system 50 will inter- 
act with a continuous media storage and retrieval sys- 
tem 70 to retrieve the continuous media objects stor- 
ed by an associated file server 40. 

55 The infonmatlon retrieval system 50 is preferably 

. embodied as the Linus Database System, developed 
by AT&T, which is a "front end" system that allows a 
workstation to access information over a networi< that 
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Is stored in a number of distributed databases. For a 
description of the Linus Database System, see Lorin- 
da L Cherry and Robert K. Waldstein, 'Electronic Ac- 
cess to Full Document Text and Images through Li- 
nus," AT&T Technical Journal. Vol. 68, No. 4, pp. 72- 
90 (July/August 1989). Incorporated herein by refer- 
ence. 

An information retrieval system 50, such as the 
Linus Database System, typically provides the ability 
to store, index, search, and retrieve information from 
a plurality of databases. In addition, such information 
retrieval systems 60 typically provide an authentica- 
tion mechanism that ensures that stored Infonmation 
is viewed.only by authorized users. 

The §iill image storage and retrieval system 60 is 
preferably embodied as the Ferret Document Brows- 
ing System, developed by AT&T, which together with 
file server 35, provides storage and retrieval func- 
tions for a collection of databases, such as databases 
80, 85, 90, containing still images, such as images of 
documents, drawings and photographs. For a de- 
scription of the Ferret Document Browsing System, 
see Howard P. Katseff and Thomas B. London, "The 
Ferret Document Browser," Proa USENIX Summer 
1993 Technical Conference, Cincinnati, OH. (June 
1993). incorporated herein by reference. 

Alternatively, the information retrieval system 50 
and the still iniage storage and retrieval system 60 
may be embodied as a single integrated system, such 
as the Document Management S^tem, commercially 
available from NCR Corp., or the RightPages Image- 
Based Electronic Library System, developed by AT&T 
Bell Laboratories. For a description of the RightPages 
Image-Based Electronic Library System, see Guy A. 
Story et al., "The RightPages Image-Based Electron- 
ic Library for Alerting and Browsing," Computer, 17- 
26 (September 1992); Lawrence O'Gorman, "Image 
and Document Processing Techniques for the Right- 
Pages Electronic Library System," Proa 1 1 th Interna- 
tional Conference on Pattern Recognition. The Neth- 
erlands, 260-63 (August 30-September 3, 1992). 
each incorporated herein by reference. 

The still image storage and retrieval system 60 
preferably Includes a still image infonmation capture 
subsystem 210, discussed further below relative to 
FIG. 2, which Is utilized to obtain an electronic repre- 
sentation of each still image object and to store the 
electronic representation on a file server, such as file 
server 35. 

The still image objects are preferably stored by a 
still image file server, such as file server 35, on a plur- 
ality of disk arrays or other secondary storage devic- 
es. The still Image file server 35 may be embodied as 
a Sun SPARC workstation, commerciany available 
from Sun Microsystems, Ina. of Mountain View. CA. 
Alternatively, the still image objects may be stored on 
other mass storage devices that allow stored data to 
be accessed and shared by a plurality of workstations 
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15. 25, 30 over a network 20. In a preferred embodh 
ment, file server 35 can be embodied as a plurality of 
distributed file servers which nnay be positioned at a 
plurality of remote locations throughout network 20. 

The continuous media storage and retrieval sys- 
tem 70, together with file server 40, provides storage 
and retrieval functions for a collection of databases, 
such as databases 95, 100, containing continuous 
media, such as video, audio and aniinated graphics. 
10 For a discussion of an Illustrative continuous media 
storage and retrieval system 70. see Lawrence A. 
Rowe and Brian C. Smith, "A Continuous Media Play- 
er." Proc 3d Inn Workshop on Network and Operat- 
ing System Support for Digital Audio and Video, San 
1$ Diego, CA. (Nov. 1 992), incorporated herein by refer- 
ence. The Continuous Media Player system Is avail- 
able from the Univereity of California, Bericeley. 

The continuous media storage and retrieval sys- 
tem 70, such as the Continuous Media Player system 
20 referenced above, preferably provides the mecha- 
nism for digitizing and compfessing audio, video and 
other continuous media for storage by a number of 
distributed file servers. In a manner described further 
below. In addition, the continuous media storage and 
25 retrieval system 70 provides the mechanism for ac- 
cessing stored continuous rhedia data files over net- 
work 20 for presentation on a workstation, such as 
workstation 15. 

Alternatively, the contiriuous media storage and 
30 retrieval system 70 may be embodied as the multime- 
dia network system commercially available from Ru- 
ent. Inc.. now called Novell Multimedia. 

The continuous media storage and retrieval sys- 
tem 70 preferably includes a continuous media infor- 
ms malion capture subsystem 310, discussed further be- 
low relative to FIG. 3, which Is utilized to obtain ah 
electronic representation of each continuous media 
object and to store the electronic representation on a 
file server, such as file server 40. 
40 The continuous media objects are preferably 
stored by a continuous media file server, such as file 
server 40, on a plurality of disk arrays or other sec- 
ondary storage devices. For a discussion of illustra- 
tive continuous media file seryere, see Roger L Has- 
45 kin, "The Sharic Continuous-Media RIe Server," Proc. 
IEEE COMPCON '93, pp. 12-15, San Francisco. CA. 
(Feb. 1993); Fouad A. Tobagi & Joseph Pang, "Star- 
Works ""*-A Video Applications Server." Proc. IEEE 
COMPCON '93, pp. 4-11, San Francisco, CA. (Feb. 
so 1 993). In a preferred embodiment, file server 40 can 
be embodied as a plurality of distributed file servere 
which may be positioned at a plurality of remote loca- 
tions throughout network 20. 

In a preferred embodiment, each multimedia ol>- 
55 ject stored in the multimedia system 1 0 is indexed by 
a content profiler 75 that acquires bibliographic Infor- 
mation on each multimedia object for storage In a pro- 
file database 105. as discussed further below. 

4 
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- As illustrated In FIGS. 2 and 3. multimedia ob- 
jects are stored by the networked multimedia system 
1 0 by capture processes that obtain an electronic rep- 
resentation of each multimedia object Multimedia in- - 
formation that Is received In an electronic format, 
such as a document received directly from a comput- 
ing system in electronic form, may be entered directly 
in the system 10. Multimedia information that is re- 
ceived in a non-eieclronlc format, however, such as 
Information received on paper or film, must first be 
converted into an electronic format using Imaging 
technology. 

An illustrative still image information capture 
subsystem 21 0 for use by the still Image storage and 
retrieval system 60 is shown In FIG. 2. The still image 
Information capture subsystem 210 will convert a 
printed document 215 or diagram 220 Into an elec- 
tronic format using a document scanner 225, which 
will convert a printed page 215, 220 Into an electronic 
format by making a bitmap image of the page. Simi- 
larly, a photograph 230 can preferably be converted 
Into an electronic forniat for entry Into the system 10 
using a grayscale scanner 235. 

The electronic format images produced by scan- 
ner 225 and grayscale scanner 235 preferably con- 
form to the well-known Tag Image File Format (TIFF) 
specification. The scanned Images are preferably 
compressed by compression circuitry 245, 250, re- 
spectively, priorlo storage by file server 35. 

Multimedia Infonnation that is received in an 
electronic form, such as documents that are received 
directly from a source computer 255 in electronic 
form, can be stored directly by file server 35 following 
compression by compression circuitry 260. 

The textual portions of a document 215 or a dia- 
gram 220 are preferably converted Into a computer- 
readable format by an optical character recognition 
system 240 for transfer to the content profiler 75 via 
data link 270. Similarly, a copy of each document re- 
ceived directly from a source computer 255 in elec- 
tronic computer-readable fonn Is also preferably 
transferred to the content profiler 75, via data link 
275. As discussed further below, the computer-read- 
able form of a inultimedia object may be utilized by 
the content profiler 75 for the automatic generation of 
, an object profile, which may consist of an abstract 
and other bibliographic Information. 

An illustrative continuous media information cap- 
ture subsystem 310 for use by the continuous media 
storage and retrieval system 70 Is shown in FIG. 3. 

An analog video Input module 315 Is preferably 
configured to receive analog video signals via data 
link 320 from, e.g., a video cassette recorder 325 
and/or a video camera 330, as shown In FIG. 3r. Sinv 
ilarly, an analog audio input module 340 is preferably 
configured to receive analog audio signals via data 
link 345, from, e.g., an audio tape recorder 350 and/or 
a microphone 355. Video camera 330 and micro- 



phone 355 may be positioned during a presentation, 
such as a meeting or seminar, to capture the visual 
and audio portions of the presentation, respectively. 
In this manner, analog video and audio signals 

5 generated during a presentation by camera 330 
and/or microphone 355, respectively, may be re- 
ceived by input modules 315, 340 for storage in the 
multimedia system 10. 

Preferably, the analog video and audio signals re- 

10 ceived by Input modules 315, 340 are processed by 
a digitizer/compressor 360, prior to storage by the file 
server 40. The digitizer/compressor 360 may be em- 
bodied as a Parallax XVideo System, consisting of a 
board having a JPEG CODEC chip and associated 

15 software, conrunercially available from Parallax 
Graphics, Inc. of Santa Clara. CA. The digitizer/conv 
pressor 360 will receive analog video and audio in- 
puts from input modules 31 5, 340. The digitizer/conrv 
pressor 360 will preferably digitize and compress the 

20 received video input and digitize the received audio 
Input 

The data stream generated by the digitbier/conv 
pressor 360 Is preferably In the JPEG Movie File for- 
mat, which is described in 'JPEG Movie File Specifi- 

25 cation. Release 1.0," Parallax Graphics, Inc., Santa 
Clara, CA. (Nov. 5, 1992), Incorporated herein by ref- 
erence. The JPEG Movie File format Interleaves one 
frames' worth of audio with a frame of video. For ex- 
ample, if video signals are being stored for a display 

30 rate of 1 0 frames per second (fps), theresuiting inter- 
leaved data file will include a repeating pattern of one 
frame of video, followed by a tenth of a second's 
worth of audio. 

The JPEG Movie File format is preferred because 

35 of the inherent synchronization of the audio and video 
streams that results from the paired storage of a vid- 
eo frame with its associated audio, as discussed fur- 
ther below. In addition, since the JPEG technique 
compresses each video frame Independently, there 

40 Is a resulting frame independence that allows a video 
frame to be easily dropped in the event of network 
congestion, as discussed below. Other video com- 
pression schemes may be utilized, such as the MPEG 
compression standard. However, the MPEG com- 

45 pression technique, which takes into account the sirrv 
llarity between successive Images, is less prefenred 
because the resulting frame dependence makes It 
more difficult to drop video frames in the event of net- 
work congestion. 

so It is noted that typical workstations, such as the 
Sun woricstations discussed above. Include audio 
hardware 23 for receiving analog audio inputs and 
storing the audio signals in a digitized format Thus, 
where analog audio signals are being processed, 

55 without any associated video, a workstation, such as 
a Sun workstation, can receive the analog audio sig- 
nal via Its audio input port, and then digitize the audk) 
signal for storage as a multimedia object by a file ser- 
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ver, such as the file server 40. 

An electronic blackboard 358 is preferably pro- 
vided, so that annotations niade by a speaker on the 
blackboard 358 during a presentation may be eleo- 
tronically captured for storage In the mulOmedia infor- 5 
mation system 10. The digital signals generated by 
the electronic blackboard 358 can be stored directly 
by file server 40 in an electronic file following com- V . 
pression by compression circuitry 372. 

A digital audioAvideo input module 370 is prefer- 10 
ably configured to receive digital signals from a plur- 
ality of digital audio or video sources, such as a 
source 36$, which may be a digital videodisk player. 
The digita{:«ignals received by input module 370 can 
be stored directly by file server 40 following compres- is 
compression circuitry 375. 
Other continuous multimedia infonmatfon that is 
received in an electronic form directly from a source 
computer 380. such as animated graphics, can be 
stored by ..file server 40 following compression by 20 
compression circuitry 385, 

Once the multimedia infomiatlon has been stored 
in an electronic format by the information capture 
subsystems 210, 310 in the manner described above, 
the resulting multimedia object may be selected by a 25 
user and then transmitted over the network 20 to a 
workstation 15. A workstation, such as the worksta- 
tion 1 5, will receive the transmitted multimedia object 
for presentation to the user. 

In a preferred embodiment, the content profiler 30 
^5 will create an object profile 400. shown illustrative- 
ly in FIG. 4, for each multimedia object Each object 
profile 400 includes bibliographic infonmation that 
may be used for Indexing and searching each multi- 
media object In a preferred embodiment, each object 35 
profile 400 is stored centrally in a profile database 
1 05 associated with the content profiler 75, as shown 
in FIG. 1. 

The object profile 400 can include a plurality of 
entries for listing bibliographic Infonmation that de- 40 
scribes relevant aspects of each multimedia object, 
such as a title entry 410, an author or speaker entry 
420, an abstract entry 430, a date entry 440 and a 

keywordentry450forlistingthekeywordsassodated 
with the multimedia objert. i . 4$ 

In addition, each object profile 400 preferably In- 
cludes an entry 460 containing a pointer to the mem- 
ory location where the multimedia object is stored. 
I.e„ an indication of the particular file server, such as 
file server 35 or 40, which controls the storage of the 50 
multimedia object In one embodiment of the inven- 
tion, discussed further below, entry 460 can be a mul- 
ti-valued entry containing pointers to a number of re- 
lated multimedia objects and/or data files. 

As indicated above, the content profiler 75 can .55 
automatically generate object profiles, such as the 
object profile 400, for those multimedia objects that 
are stored in a computer-readable fonm. The content 



profiler 75 will receive multimedia objects -in a com- 
puter-readable form via data link 270 from the optical 
character recognition system 240 and via data link 
275 from source computer 255, as illustrated in FIG. 
2. The object profiles associated with multimedia ob^ 
jects that are not stored in a computer-readable fonti 
however, such as the object profiles for photographs! 
audio and video, can be generated manually for entry 
in the content profiler 75. 

The information retrieval system 50 allows a user 
to access the multimedia infonmation that has been 
stored in the plurality of databases, such as databas- 
es 80, 85. 90, 95. 100. assodated with file servers 35, 
40. As discussed above, each multimedia object has 
an associated object profile, such as the object pro- 
file 400. that b stored in a profile database 105. The 
infonmation retrieval system 50 preferably provides a 
mechanism for searching the plurality of object pro- 
files stored in the profile database 105 to Identify 
those multimedia objects that saUsfy a user- 
specified query. 

A user may employ a workstation, such as the 
woricstation 15. to enter a query specifying parame- 
ters for retrieving multimedia objects from^the multi- 
media system 1 0. The infonmation retrieval system 50 
will retrieve all the object profiles from the profile da- 
tabase 105 that con-espond to multimedia objects 
that satisfy the user-specified query. Thereafter, the 
/ Infomiation retrieval system 50 will transmit the ob- 
ject profiles satisfying the entered query to the woric- 
stalten 15. The user can review those object profiles 
that satisfy the entered query and select those mul- 
timedia objects that should be downloaded to the 
workstation 15. 

Once a multimedia object has been selected by 
a user, the information retrieval system 50 will access 
the entry 460 In the object profile that indicates the 
memory locations where the selected multimedia ob- 
jects are stored. The infonmation retrieval system 50 
will send a retrieval message over network 20 to the 
respective storage and retrieval system 60. 70 which 
accesses the appropriate file server 35, 40 indicated 
by the retrieved pointer. The compressed file contain- 
ing the requested multimedia object is transmitted 
from the respective file server to the workstation 15, 
in a known manner. 

If the retrieved multimedia object is a stiil image 
object vyith many pages, such as a document, the 
woricstation 15 will thereafter decompress each page 
of the multimedia object as requested for display by 
the user. In a known manner. 

If, however, the retrieved multimedia object Is 
comprised of continuous media, such as audio or vid- 
eo, the entire multimedia object will not nonmally be 
retrieved at once, due to the memory limitations as- 
sociated with many workstations. Rather, the woric- 
station 15 will utilize a prefetch routine to retrieve sev- 
eral frames worth of audio and video from the respec- 
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tive file server for storage in an audio buffer 110 and 
a video buffer 115, respectively, as described further 
below. Accordingly, the received video frames will be 
decompressed by the workstation 15 as they are re- 
ceived over the network 20 from the respective file 
server, in a known manner. 

For a multimedia object that contains a plurality 
of media, such as the audio and video portions of a 
recorded presentation, the workstation 15 will prefer- 
ably have a mechanism for synchronizing the presen- 
tation of the various outputs. In the preferred embodi- 
ment discussed above, a multimedia object contain- 
ing the audio and video portions of a recorded presen- 
tation will be stored as a data file containing inter- 
leaved audio and video Information In the JPEG Mov- 
ie File format 

The workstation 15 can process the interleaved 
JPEG data file, having a frame of video paired with Its 
associated audio, and direct the audio information. In 
real time, to the audio hardware 23 of the workstation 
15, In a known manner, while the video Information Is 
decompressed utilizing a software routine for presen- 
tation on display 510, In a known manner. 

Alternatively, the multimedia system 10 can im- 
plement known synchronization algorithms to control 
the recording and playback of continuous media. Dur- 
ing playback, the synchronizatton algorithm monitors 
the Internal dock of the workstation, such as worksta- 
tion 15, to detemfiirie vyhich iaudio and video frames 
should be played and makes adjustments, if neces- 
sary. 

According to one aspect of the invention, the 
above networked multimedia system 10 is utilized to 
record and store multimedia presentations, such as 
seminars, meetings or conferences, together with any 
supplemental materials that may be referenced dur- 
ing the presentation, such as viewgraphs, slides, 
handouts or blackboard annotations. The recorded 
presentation, together with the stored supplemental 
. materials, may thereafter be accessed over networic 
20 for presentation to a user on a workstation, such 
as the workstation 15. 

The audio and video portions of the presentation 
are preferably captured and converted into a multime- 
dia object using the continuous media Infonmation 
capture subsystem 310, in the manner described 
above relative to FIG. 3. The supplemental materials 
that consist of still images, l.e., viewgraphs, slides 
and handouts, are preferably separately processed 
from the recorded presentation and stored as a sep- 
arate multimedia object by the still image Infonmation 
capture subsystem 210, In the manner described 
above relative to FIG. 2. 

In addition, according to a further feature of the 
Invention discussed further below, the supplemental 
materials that consist of continuous media, l.e.. the 
annotations made by a speaker on an electronic 
blackboard 358, are preferably stored as a separate 



multimedia object by the continuous media infonma- 
tion capture subsystem 310, In the manner described 
above relative to FIG. 3. 

In a preferred embodiment. Illustrated in FIG. 5, 

5 the recorded presentation, together with the stored 
, supplemental materials, may be simultaneously pre- 
sented on a display, such as display 510 of worksta- 
tion 15, containing two separate viewing windows 
520, 530. In this manner, the appropriate supplemen- 

10 tal materials may be displayed in window 520 as they 
are referenced during the recorded presentation be- 
ing displayed In window 530. 

Preferably, the presentation of the audio and vid- 
eo components of the recorded presentatton Is syn- 

15 chronized with the presentatbn of the supplemental 
materials In window 520. As discussed further below, 
the supplemental materials are an additional medium 
to be synchronized with the presentation of the audio 
and video portions of the recorded presentation. 

20 As is known, a video presentation consists of a 
large number of sequential video frames which may 
be recorded at a fixed rate, such as a frame rate of 
1 0 frames per second. Thus, the video frame number 
can be utilized as an index Into the video stream. In 

25 addition, as discussed further below, the supplemen- 
tal nwiterials that are associated with the recorded 
presentation, such as the viewgraphs. may be ident^ 
fled with the video frames In which they are refer- 
enced. 

30 ' 

SYNCHRONIZATION OF THE RECORDED 
PRESENTATION AND STILL IMAGE 
SUPPLEMENTAL MATERIALS 

35 Where the supplemental materials associated 
with the recorded presentation is comprised of a mul- 
ti-page still Image object, such as viewgraphs and 
handouts, each page of the still Image multimedia ob- 
ject may be numbered sequentially In the order in 

40 which it is referenced during the recorded presenta- 
tion. The recorded presentation and the associated 
still Image supplemental materials may thereafter be 
synchronized by means of a still image hyperlink file, 
such as the stilllmage hyperlink file 600 illustrated In 

45 FIG. 6. 

The still image hyperlink file 600 preferably con- 
sists of a plurality of video frame-still Image page 
pairs, Le., the page number of the still image multime- 
dia object in column 620 together with the corre- 

50 spending frame number in column 610 In which the 
still image page Is first referenced. 

The still image hyperlink file 600 will include a 
plurality of rows, such as rows 625, 630, 635. 640. 
642. 644. 646, each corresponding to a video frame- 

55 still image page pair. Each time a page of the still in>- 
age supplemental material object, such as a view- 
graph. Is referenced during the presentation, an entry 
is made in column 620 of the still Image hyperlink file 
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600 with the appropriate page number of the supple- 
mental material, together with the number in column 
610 of the video frame in which the supplemental ma- 
terial page is first referenced. If the same viewgraph, 
, , or other still image supplemental material. Is refer- 5 

enced more than once during a recorded presenta- 
!• . tlon, It is preferably assigned a unique page number 

; . . - for each appearance of the stIU image page. 

According to a feature of the invention, a pair of 
n independent processes, which may interact by means io 

of a message passing mechanism, are preferably util- 
ized to display a recorded presentation and its asso- 
^ elated still image supplemental materials. A video 
process controls the presentation of the audio and 
video portions of the recorded presentation. The vid- is 
eo process preferably incorporates the features and 
functions of the presentation processes of the Contin- 
uous Media Player, described in the reference incor- 
porated above, as well as the additional features 
and'functions descn*bed below, 20 
Where the recorded presentation has associated 
. still image supplemental materials, the video process 
will Interact with a still image presentation process in 
order to present the appropriate stiil image supple- 
mental materials in the window 520 of display 510. 25 
The still image presentation process preferably Incor- 
porates the features and functions of the still image 
presentation processes of the Ferret Document 
Browser, -.jdescribed in the reference incorporated 
above, as well as the additional features and funo- 30 
, . -tions described below. 

The video process preferably utilizes a still image 
synchronization subroutine, illustrated In FIG. 7, to 
synchronize the still image supplemental materials in 
window 520 with the ongoing video presentation in 35 
window 530. If the recorded presentation has assod- 
ated still Image supplemental materials, the video 
process will enter the still Image synchronization sub- 
routine at step 700 as each video frame is presented 
in window 630, as shown in FIG. 7. 40 

During step 710. the still Image synchronization 
subroutine will access the stiil image hyperlink file 
600 to ctetemiine if the appropriate page of the still im- 
age file Is being displayed in window 520. The still Im- _ , : . 
age synchronization subroutine will access the row of 45 
the still Image hyperlink file 600 having the largest 
video frame number that is less than or equal to the 
frame number currently being displayed. 

Once the appropriate row of the still image hyper- 
link file 600 is accessed, the still image synchronize- so 
lion subroutine will retrieve, during step 720, the ap- 
propriate still image page number from the wrre- 
sponding entry In column 620. The still image syn- 
chronization subroutine vyill compare during step 730 
the stiil image page number retrieved from the still 55 
image hyperlink file 600 in the prevtous step to the 
page number of the still image currently displayed. If 
it b detennined during step 730 that the retrieved still 



Image page number does equal the currently dis- 
played still image page number, the appropriate still 
image is being displayed, and the still Image syn- 
chronization subroutine will be exited at step 760 
Process control will thereafter return to the video 
process. 

If, however, it is determined during step 730 that 
the retrieved still Image page number does not equal 
the cunrenUy displayed still Image page number, the 
synchronizaUon subroutine will send a message to 
the still image presentation process during step 750, 
which includes an Indicatton of the still image page 
number which should be displayed in window 520, as 
retrieved during step 720. The still image synchroni- 
zation subroutine will thereafter be exited during step 
760 and process control will return to the video proc- 
ess. 

When the still image presentation process re- 
ceives the still image page number from the still Im- 
age synchronization subroutine, transmitted during 
step 750, the still image presentation process will de- 
compress the appropriate page of the supplemental 
materials multimedia object corresponding to the re- 
ceived still image page number for presentation in 
window 520, in a known manner. 

SYNCHRONIZATION OF THE RECORDED 
PRESENTATION AND CONTINUOUS MEDIA 
SUPPLEMENTAL MATERIALS 

According to a further feature of the invention, 
the supplemental materials associated with a record- 
ed presentation may consist of continuous media, 
such as the annotations made by a speaker on an 
electronic blackboard 358 during the associated pre- 
sentation. Preferably, the electronic file that stores 
the blackboard annotations or other continuous me- 
dia supplemental materials associated with a record- 
ed presentation is martced with a time stamp or other 
counter which may serve as an index Into the record- 
ed blackboard annotations. In this manner, recorded 
blackboard events, l.e„ the placement of strokes on 
the electronic blackboard 358, can be presented with 
the video frames in which the blackboard event is ref- 
erenced. . 

The recorded presentation and associated con- 
tinuous media supplemental materials are preferably 
synchronized by means of a continuous media hyper- 
link file, such as the continuous media hyperlink file 
800 illustrated in FIG. 8. The continuous media hy- 
perlinkfile 800 preferably consists of a plurality of vid- 
eo frame-event maricer pairs, Le., a marker that iden- 
tifies the position of the recorded blackboard event 
within the associated electronic file in column 820 to- 
gether with the conresponding frame number in col- 
umn 810 in which the corresponding blackboand 
event is first referenced. 

The continuous media hyperlink file 800 will in- 
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dude a plurality of rows, such as rows 830. 840. 850. 
860, 870, each corresponding to a video frame-event 
marker pair. For each blackboard event, or stroke, 
that Is placed on the electronic blackboard 358 during 
the presentatton, an entry Is preferably made in col- 
umn 820 of the continuous media hyperlink file 800 
with the event marker that Wentifies the position of 
the recorded blackboard event within the associated 
supplemental material electronic file. In addition, the 
video frame number In which the blackboard event is 
first referenced Is placed In column 810. 

When the recorded presentation has associated 
continuous media supplemental materials, the video 
process, discussed above, will preferably Interact 
with a continuous media presentation process in or- 
der to coordinate the presentation of the appropriate 
continuous media supplemental materials In the win- 
dow 520 of display 510. The continuous media pre- 
sentation process preferably incorporates the fea- 
tures and functions of the presentation processes of 
the Continuous Media Player, in a manner similar to 
the video process, as well as the addittonal features 
and functions described below. 

The video process preferably utilizes a continu- 
ous media synchronization subroutine, illustrated In 
FIG. 9. to synchronize the continuous media supple- 
mental materials in window 520 with the ongoing vid- 
eo presentatton In window 530. jf the recorded pre- 
sentation has associated continuous media supple- 
mental nvaterials, the video process will enter the con- 
tinuous media synchronization subroutine at step 900 
as each video frame is presented in window 530, as 
shown In FIG. 9. 

During step 910, the continuous media synchron- 
izatton subroutine will access the continuous media 
hyperlink file 800 to determine if the appropriate seg- 
ment of the electronic file, i.e., the appropriate black- 
board event. Is being presented in window 520. The 
continuous media synchronizatton subroutine will ac- 
cess the row of the continuous media hyperlink file 
800 having the largest video frame number that is less 
than or equal to the frame number currently being dis- 
played.' 

Once the appropriate row of the continuous me- 
dia hyperiinkf lie 800 is accessed, the continuous me- 
dia synchronization subroutine will retrieve, during 
step 920, the appropriate event marker from the cor- 
responding entry In column 820. During step 930, the 
continuous media synchronization subroutine will 
compare the event marker retrieved from the contin- 
uous media hyperlink file 800 during step 920 to the 
event marker associated with the blackboard event 
currently being displayed. 

If it is detemiiped during st;ep 930 that the re- 
trieved event marker equals the event maricer asso- 
ciated with the blackboard event currently being dis- 
played, the appropriate blackboard event Is being dis- 
played, and the continuous media synchronization 
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subroutine wilt be exited at step 960. Process control 
will thereafter return to the video process. 

If, however, it Is determined during step 930 that 
the retrieved event marker does not equal the event 
5 marker associated with the blackboard event current- 
ly being displayed, the continuous media synchroni- 
zation subroutine will send a message to the contin- 
uous media presentation process during step 950, . 
which Includes the event marker as an Indication of 
10 the file position from which the associated electronic 
file that contains the recorded blackboard annota- 
tions should be restarted. The continuous media syn- 
chronization subroutine will thereafter be exited dur- 
ing step 960 and process control will return to the vid- 
15 eo process. 

When the continuous media presentation proc- 
ess receives the event marker from the continuous 
media synchronization subroutine, transmitted dur- 
ing step 950. the continuous media presentation proc- 
20 ess will make an adjustment to the presentation of the 
continuous media supplemental materials in window 
520 by restarting the presentation from the position 
In the electronic file indicated by the received event 
mari(er. in a known manner. 
25 When a recorded presentation has associated 
still image and/or continuous media supplemental 
materials, a single object profile, similar to the object 
profile 400 illustrated in FIG. 4, is preferably utilized 
to profile the multimedia objects corresponding to the 
30 recorded presentation and associated supplemental 
materials. The entry 460 of object profile 400 is pre- 
ferably a multi-valued entry containing a plurality of 
pointers, e.g., a pointer to the multimedia object cor- 
responding to the recorded presentation, a pointer to 
35 the multimedia object corresponding to the supple- 
mental materials and, where appropriate, a pointer to 
the hyperlink file 600 associated with the recorded 
presentation. 

Thus, when a user selects a recorded presenta- 
40 tlon having associated supplemental materials, in the 
manner described above, the Information retrieval 
system 50 will access the entry 460 to retrieve the 
plurality of pointers. Thereafter, the Informatton re- 
trieval system 50 will siend a retrieval message over 
45 networic 20 to each file server Indicated by the plur- 
ality of retrieved pointers. The data files containing 
the recorded presentation, corresponding supple- 
mental materials, and, where appropriate, the respec- 
tive hyperlink file 600, 800, will be transmitted by the 
so respective file servers to the workstation, such as 
workstation 15. 

Window 530 of display 51 0 Includes a frame nunv 
ber scroll bar 540 that allows a user to scroll through 
video frames. Once the user has selected a starting 
55 frame using the frame number scroll bar 540, the vid- 
eo process will restart the video, together with the 
corresponding audio, from the selected video frame. 
If the recorded presentation has associated still im- 
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age supplemental materials, the video process will 
utilize the still image synchronization subroutine of 
FIG. 7 to retrieve the appropriate stin image page 
number from the still Image hyperlink file 600 con^e- . 
spending to the selected starting video frame, for s 
transmittal to the still image presentation process. 

Similarly. If the recorded presentation has asso- 
cialed continuous media supplemental materials, the 
video process will utilize the continuous media syn- 
chronizaUon subroutine of FIG. 9 to retrieve the ap- io 
propriate event marker from the continuous media hy- 
perlink file. 800 corresponding to the starting video 
frame selopted using the frame number scroll bar 
.y. 540. for transmittal to the continuous media presen- 
. tation process. 

It is noted that If a user has selected a frame num- 
ber that has not yet been received by the workstation 
16 from the respectwe file server, the video process 
must first transmit a retrieval request to the file server 
to retrieve the necessary frames. 20 

In addition, window 530 includes a playback 
speed scroll bar 550 that allows a user to control the 
playback speed, in frames per second, of the record- 
ed presentation. In a known manner. The playback 
speed scroll bar allows the user to adjust the playback 25 
speed from a minimum of zero frames per second. 
I.e., a still image, up to the maximum recorded frame 
rate of the video, In either forward or reverse mode, 
Pnce the user has selected a video playback speed 
using the playback speed scroll bar 550. the video 30 
.^process will adjust the rate of data being requested 
from the storage and retrieval system 70 to the select- 
ed playback speed, in addition to making local adjust- 
ments to the video and audio outputs of workstation 

35. 

It Is noted that where the recorded presentation 
has associated still Image supplemental materials, 
the video process will utilize the still Image synchron- 
ization subroutine of FIG. 7 to retrieve the appropriate 
still image page number from the still image hyperlink 40 
file 600 In the same manner for both forward and re- 
verse playback of the video, as described above. Sim- 
ilarly, where the recorded presentation has assodat- 
ed continuous media supplemental materials, the vid- 
eo process will utilize the continuous media syn- 4S 
chronization subroutine of FIG. 9 to retrieve the ap- 
propriate eventmarker from the continuous media hy- 
perlink file 800 In the same manner for both forward 
and reverse playback of the video, as described 
above. ^ 

In a preferred emlK)diment, a user can manipu- 
late the still Image or continuous media supplemental 
materials in window 520 to control the video playback 
In window 530. In this manner, the presentation of the 
supplemental materials can serve as an index to the 55 
recorded presentation. A user can proceed through 
the supplemental materials In window 520. e.g., by 
clicking the leift button of a mouse in window 520 to 



move through the supplemental materials In a for- 
ward direction or clicking the rig ht button of the mouse 
to move In a reverse direction. 

Once the user has selected a desired still Image 
page or continuous media event In window 520, the 
recorded presentation In window 530 can be restarted 
from the video frame where the still Image page or 
continuous media event Is referenced in the recorded 
presentation. Upon selection of a desired page or 
event In window 520. the still Image presentation 
process or the continuous media presentatron proc 
ess, as appropriate, preferably sends a message to 
the video process with the selected still image page 
number or continuous media event maricer. The video 
process will thereafter access the respective hyper- 
link file 600, 800 to retrieve the video frame that cor- 
responds to the selected still Image page or continu- 
ous media event The video process will then restart 
the video from the retrieved frame number. 

It Is noted that where the supplemental materials 
associated with a recorded presentation consist of a 
combination of still images and continuous media, a 
third viewing window (not shown) may be provided on 
display 510 for the simultaneous presentation of the 
recorded presentation, together with the stored still 
image supplemental materials and continuous media 
supplemental materials. Alternatively, window 520 
may be configured to alternately present still Images 
or continuous media, as they are referenced during 
the recorded presentation, in a known manner. 

NETWORK CONGESTION 

In order to compensate for congestion on net- 
woric 20 that causes the delayed arrival of audio and 
video data, the video process will preferably prefetch 
audio and video frames from the respectwe file ser- 
ver 40 for storage in audto and video buffere 110, 115, 
respectively, on the user's workstation to minimize 
the effect of networic congestion on the playback of 
the recorded presentation. Since two-way video con- 
ferencing applications demand rapid response times, 
the buffering of audio and video data would not nor- 
^. mally be tolerated In such applications. It Is noted, 

however, thatduring the playbadc of recorded presen- 
tations the buffering of audio and video data would 
not be detected by a user. 

According to one feature of the invention, the net- 
woriced multimedia system compensates for conges- 
tion on network 20 using an adaptive control algo- 
rithm to dynamically vary the rate at which video 
frames are retrieved from the respective file server 
40 over networic 20, In response to increases or de- 
creases in the amountof other data being transmitted 
across the network 20. For a discussion of an illustra- 
tive adaptive control algorithm, see section 2.3 of the 
Continuous Media Player reference. Incorporated 
above. 
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In order to maximize the playback quality of the 
recorded presentation, the audio component of the re- 
corded presentation Is preferably given preference 
over the video component It has been found that a 
person viewing a recorded presentation will object 
more strongly to defects in the audio portion of the re- 
corded presentation than to defects in the video por- 
tions of the presentation. 

In a preferred embodiment, the video process 
utilizes a data buffer monitoring subroutine, illustrat- 
ed in FIG. 10, to maintain a pre-defined amount of au- 
dio and vWeo data in the audio and video buffers 110, 
115. The data buffer monitoring subroutine will con- 
tinuously monitor the audio and video buffers 110, 
115 during step 1 000. until the amount of audio or vid- 
eo data stored In the audio or video buffers 110, 115 
drops below a predefined threshold value, as detect- 
ed by step 1010. 

If it is determined during step 1010 that the 
amount of audio or video data In the audio or video 
buffers 110, 115, respectively, has fallen below the 
. desired threshold value, the data is not aniving at the 
woritstatlon 15 from the respective file server 40 over 
network 20 as fast as it is being presented to a user 
by the workstation 15. Thus, the data buffer monitor- 
ing subroutine will reduce the requested video play- 
back rate during step 1015 by requesting that the file 
server 40 transmit fewer video frames per second to 
the workstation 15. It Is noted, however, that the data 
buffer monitoring subroutine will preferably continue 
to request all of the audio data from the file server 40. 

After the requested video playback rate is re- 
duced during step 1015, a test is performed during 
step 1 020 to detennine If the amount of audio or video 
data in the audio or video buffers 110, 115 is still be- 
low the desired threshold value, tf it Is determined dur- 
ing step 1020 that the anrount of audio or video data 
in the audio or video buffers 110, 115 has risen above 
the desired threshold value, program control will pro- 
ceed to step 1060, discussed below. 

If, however, it is detemiined during step 1020 that 
the amount of audio or video data in the audio or video 
buffers 110, 115 is still below the desired threshold 
value, a test Is perfonmed during step 1025 to deter- 
mine If the requested video playback rate has been 
. reduced to the minimum value, l.e., a playback rate of 
0 frames per second (fps). 

If it Is determined during step 1025 that the re- 
quested video playback rate has not yet been reduced 
to the minimum value, program control will return to 
step 1 01 5 for a further reduction In the requested vid- 
eo playback rate. 

If it Is determined during step 1025 that the re- 
quested video playback rate has been reduced to the 
minimum value, network congestton conditions are 
so extreme that even though no video data is being 
transmitted across the network 20, the audio data is 
still not arriving fast enough over the network 20 to 



maintain the desired size of the audio buffer 110. In 
a preferred embodiment, the data buffer monitoring 
subroutine will compensate for the delayed arrival of 
audio data by playing the audio data from the audio 

5 buffer 110 at slower than real-time. 

Thus, the data buffer monitoring subroutine will 
begin playing the audio at a reduced speed during 
step 1030. The data buffer monitoring subroutine will 
continue playing the audio at the reduced speed until 

10 It Is detennined during step 1040 that the amount of 
audio data In the buffer has returned to the desired 
threshold value. Once it Is determined during step 
1040 that the amount of audio data In the buffer is 
greater than or equal to the desired threshold value, 

IS the data buffer monitoring subroutine will resume 
playing the audio ata nonmal, or real-time, speed dur- 
ing step 1045. Thereafter, program control will return 
to step 1000, and continue In the mtanner described 
above. 

20 The data buffer monitoring subroutine could play 
the audio at half-speed during step 1030. e.g., by div- 
iding each frames* worth of "buffered audio data into 
n segments and then playing each segment twice. If 
it is desired to play the audio at a speed between half 
. 25 speed and normal speed, not all of the n segments 
are played twice. Similarly, if it Is desired to play the 
audio at a speed less than half speed, some of the n 
segments may be played more than twice. 

In a prefenred embodiment, the data buffermon- 

30 itoring subroutine will gradually adjust Ihe playback 
speeds of the audio during steps 1 030 and 1 045 in or- 
der to make the transition from one speed to another 
less noticeable to a listener. For example, the data 
buffer monitoring subroutine can reduce the audio 

35 playback rate during step 1030 according to a scale 
that gradually adjusts the playback rate between a 
defined maximum and minimum audio playback rate. 
In addition, by monitoring the rate at which the audio 
buffer 110 is emptying, the data buffer monitoring 

40 subroutine can determine how quickly the audio play- 
back rate should be reduced during step 1030 and 
what the minimum audio playback rate should ulti- 
mately be. 

: In an alternate embodiment, the data buffer mon- 

45 Itoring subroutine could play the audio at a reduced 
speed during step 1 030 by utilizing a well-known pitch 
extraction process, which identifies where the paus- 
es are In the audio and makes the pauses longer. 
Once the data buffer monitoring subroutine has 
60 stabilized the amount of data in the audio and video 
buffers 110, 115, as detected during step 1020. pro- 
gram control will proceed to step 1060. During step 
1060, the amount of video data requested from the re- 
. spective file server will be gradually increased. 
55 Thereafter, program control will return to step 1000 to 
continue monitoring the audio and video buffers 110, 
115 In the manner described above. 

Although the data buffer monitoring subroutine 
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will drop video frames during times of network con- 
gestion, it is preferred that the video process will ac- 
cess the still image synchronization subroutine of 
FIG. 7 where the supplemental materials consist of. 
still images or the continuous media synchronization s 
subroutine of FIG. 9 where the supplemental materi- 
als consist of continuous media for each video frame, 
regardless of whether or not the video frame is actual- 
ly displayed in window 530. In this manner, synchron- 
ization is maintained between the presentation of the io 
supplemental materials in window 620 and the con- 
tinuing audio presentation, even in the absence of vid- 
eo frames; 

-•> It is tocbe understood that the embodiments and 
variation^hown and described herein are illustrative is 
of the prindples of this invention only and that various 
modifications may be implemented by those skilled in 
the art without departing from the scope and spirit of 
the Invention. 

20 

. Claims 

1. A system for storing and distributing a recorded 
presentation and associated supplemental mate- 
rials referenced during said recorded presenta- 
tion, saki recorded presentatfon having audio and 
. video components, said storage and distribution 
system employing a computing system having a 
processor and storage means, said storage and so 
distribution system comprising: 

means for storing said audio and video 
components of said recorded presentation in said 
storage means; 

means for storing an electronic represen- 35 
tation of said associated supplemental materials 
in said storage means; 

means for accessing said storage means 
to retrieve said stored audio and video compo- 
nents of said recorded presentation and said 40 
electronic representation of said assodated sup- 
plemental materials; 

audio output means for generating an au- 
dio output responsive to said retrieved audio 
. data; . 

• a display having a first window and a seo- 

' ond window; 

means for displaying said retrieved video 
component of sakI recorded presentation in said 
first window; and 

means for displaying said retrieved elec- 
tronic representation of said assodated supple- 
mental materials in said second window. 

2, The storage and distribution system according to 55 
daim 1, further Induding means for creating an 
object profile corresponding to said recorded pre- 
sentation and said associated supplemental ma- 
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terials, said object profile containing indexing in- 
formation on said recorded presentation and said 
associated supplemental materials. 

3. The storage and distribution system according to 
daim 2. wherein said object profile indudes one 
or more pointers to memory locations storing said 
audio and vWep components of said recorded 
presentation and said electronic representation 
of said assodated supplemental materials and 
wherein said means for accessing said storage 
means accesses said memory locations indicat- 
ed by said pointers contained in said object pro- 
file to retrieve said audio and video components 
of said recorded presentation and said electronic 
representation of said associated supplemental 
materials. 

4. A system for displaying a recorded presentation 
and an electronic representation of supplemental 
materials referenced during said recorded pre- 
sentation, said recorded presentation induding a 
video component of said recorded presentatton. 
said display system comprising: 

a first window for displaying said video 
component of said recorded presentation; 

a second window for displaying said elec- 
tronic representation of said assodated supple- 
mental materials; and 

means for synchronizing the display of 
said associated supplemental materials in said 
second window to the display of said video com- 
ponent in said first window. 

5. The display system according to daim 4, wherein 
said electronic representation of said supplemen- 
tal materials consists of still images and wherein 
said means for synchronizing said supplemental 
materials and said recorded presentation further 
indudes a data file which lists each page of said 
still image supplemental materials and the con-e- 
sponding video frame number in which each still 
image page, is first referenced. 

The display system according to daim 4, wherein 
said associated supplemental materials consist 
of continuous media and wherein said means for 
synchronizing said associated supplemental ma- 
terials and said recorded presentation further in- 
cludes a data file which lists for each event occur- ' 
ring in said continuous media supplemental ma- 
terials an eventmarker that identifies the position 
of said eventwithin said electronic representation 
of said supplemental materials and the corre- 
sponding video frame number in which said event 
is first referenced. 

A method for storing and distributing a recorded 
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presentation and associated supplemental mate- 
rials referenced during said recorded presenta- 
tion, said recorded presentation having audio and 
video components, said storage and distribution 
method employing a computing system having a 
display, a processor and storage means, said 
storage and distribution method comprising the • 
steps of: 

storing said audio and video components 
of said recorded presentation In said storage 
means; 

storing an electronic representation of 
said associated supplemental materials in said 
storage means; 

accessing said storage means to retrieve 
said stored audio and video components of said 
recorded presentation and said electronic repre- 
sentation of said supplemental material; 

generating an audio signal responsive to 
said retrieved audio data; 

aeating a first window and a second win- 
dow in said display; 

displaying said retrieved video component 
of said recorded presentation in said first window; 
and 

displaying said retrieved electronic repre- 
sentation of said supplemental materials In said 
second window. 

8. The storage and distribution method according to 
daim 7, further Including the step of creating an 
object profile corresponding to said recorded pre- 
sentation and said associated supplemental ma- 
terials, said object profile containing indexing in- 
formation on said recorded presentation and said 
associated supplemental materials. 

9. The storage and distribution method according to 
daim 8, wherein said step of creating said object 
profile further indudes the step of creating an en- 
try in said object profile having one or more poin- 
ters to memory locations storing said generated 
audio and video data and said electronic repre- 
sentation of said supplemental materials and 
wheriein said step of accessing said storage 
means further indudes the step of accessing said 
memory locations indicated by said pointers of 
said object profile to retrieve said recorded pre- 
sentation and said electronic representation of 
said supplemental materials. 

10. A method for displaying a recorded presentation 
and an electronic representation of supplemental 
materials referenced during said recording pre- 
sentation, said recorded presentation including a 
video component of said presentation, said meth- 
od employing a computing system having a dis- 
play, a processor and storage means, said dis- 
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play method comprising the steps of: 

displaying said video component of said 
recorded presentation In a first window of said 
display; 

displaying said electronic representation 
of said associated supplemental materials in a 
second window of said display; and 

synchronizing the display of said associat- 
ed supplemental materials In said second window 
to the display of said video component in said first 
window. 



11. The display method according to claim 1 0, where- 
in said associated supplemental materials con- 

15 sist of continuous media and wherein said step of 
synchronizing said associated supplemental rna- 
terials and said recorded presentation further In- 
cludes the step of accessing a data file which lists 
for each event occuning in said continuous media 

20 supplemental materials an event marker that 
Identifies the position of said event within said 
electronic representation of said supplemental 
materials and the corresponding video frame 
number in which said event Is first referenced. . 

25 

12. A method for use by a computing system receiv- 
ing audio and video data over a network for pre- 
sentation to a user, said method compensating 
for congestion on said networic which causes de- 

30 layed arrival of said audio and vkSeo data, said 
video data being transmitted over said network to 
said computing system at a requested video 
transmittal rate, said audio data being presented 
to said user at an audio playback rate, said net- 

35 work congestion compensation method compris- 
ing the steps of: 

maintaining an audio buffer for storing a 
predefined amount of said audio data received 
over said network; 

40 maintaining a video buffer for storing a 

predefined amount of said video data received 
over said network; 

monitoring said audio and video buffers to 
determine when said amount of audio or video 

45 data in said buffer falls below said predefined 
amounts; 

reducing said requested video transmittal 
rate if said monitoring step detennlnes that said 
amount of audio or video data in said buffers has 
50 fallen below said predefined amounts; and 

reducing said audio playback rate if said 
amount of audio data in said audio buffer is below 
said predefined amount of audio data after said 
step of reducing said requested video transmittal 
55 rate. 

13. The network congestion compensation method 
according to claim 12, further induding the step 

13 
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of increasing said audio playback rate once said 
amount of audio data in said audio buffer is above 
said predefined amount of audio data. 

14. The network congestton compensation method s 
according to dai m 12, wherein said step of reduc- 
ing said audio playback rate is not performed un- 
less said step of reducing said requested video 
transmittal rate has reduced said requested vid- 
eo transmittal rate to a minimum value. lo 

15- The network congestion compensatton method 
according to daim 12, wherein said step of reduc- 
ing said audio playback rate plays said audio at 
a reduced speed by dividing said audio data in is 
said audio buffer into a plurality of segments and 
playing one or more of said segments at least 
twice. 
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(57) Disclosed is a networked multimedia informa- 
tion system (10) which may be utilized to record, store 
and distribute multimedia presentation (35,40,60,70) to- 
gether with any supplemjsntal materials that may be ref- 
erenced during the presentation. The recorded presen- 
tation, together with the associated supplemental mate- 
rials, may be simultaneously presented on a display 
(51 0) containing two separate viewing windows. The ef- 
fects of network (20) congestion are minimized by 



prefetching audio and video data for storage in audio 
and video buffers (110,115). An adaptive control algo- 
rithm compensates for networic congestbn by dynami- 
cally varying the rate at which video frames are retrieved 
over the network (20), in response to network traffic con- 
ditions. The audio playback speed Is reduced if the au- 
dio data does not arrive fast enough over the network 
to maintain the desired size of the audio buffer after the 
amount of video data transmitted across the network 
has been reduced to a minimum value. 
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